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programs. Routine evaluation of use issues can result in guidance for 
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recommendations that will increase the likelihood that the 
implementation process is successful and the program is routinely 
used. (Author) 
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Abstract 

While there is a recognized need for new and better ways to train soldiers 
to fight, many training programs developed in response to this need are 
used poorly or not at all. In. part, the failure to use these training 
programs stems from the lack of established procedures for monitoring 
implementation and evaluating use. Effective monitoring can provide 
information that improves the implementation process, thereby increasing 
the use of new training programs. Routine evaluation of use issues can 
result in guidance for modifying existing programs and developing new 
programs. In this paper a framework for the Life Cycle evaluation of 
Army training programs is presented. In the Life Cycle framework it is 
recognized that evaluation issues change as the program "ages." The 
paper provides an overview of the Life Cycle framework and then focuses 
on issues important during the process of implementation. The goal is 
for the "evaluator as monitor" to take actions and make recommendations 



that will increase the likelihood that the implemeri^atiojjTprocess is 
successful and the program is routinely use^. 
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IMPLEMENTING ARMY TRAINING PROGRAMS: TRANSLATING MODEL INTO ACTION 

Army programs face many of the same problems as do programs in 
education or industry. Many programs developed in response to real 
needs fail to be implemented and most of those that are implemented are " 
modified and used quite differently th^n intended by the program's 
developer. 

We at the Army Research Institute's Monterey Field Unit have recently 
become interested in the implementation and use of Army training programs. 
Our approach to these issues has been threefold. 



(Slide 1) 



First, initiate a study of the problems which training programs face and 
must overcome if they are to be successfully implemented and used. 
Second, provide guidance which Army sponsors can use to plan the imple- 
mentation of new training programs (T. Gray, C. Roberts-Gray, & W. Gray, 
in press). Third, develop a framework for the Life"C?cle monitoring and 
evaluation of training programs. Such a framework starts with the 
process of implementation and continues, ideally, until the program 
either fails or, if successful, becomes obsolete. 

Today I will talk about the third issue, the franework for the 
Life Cycle evaluation of training programs (W. Gray, J.982). I will 

p 
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first provide an overview of the Life Cycle concept and then focus my 
discussion on implementation issues. 



We see the Life Cycle concept as filling an important gap. The 
Army has established procedures for evaluating and monitoring the 
development of training programs but has no procedures for 'monitoring 
the implementation or evaluating the use of these programs. The Life 
Cycle concept addresses these latter two issues; that is, implementation 
monitoring and use evaluation. We believe that monitoring the imple- 
mentation process can increase the use of new training programs while 
evaluation of use provides valuable information for modifying and 
developing new programs. ' 

A Life Cycle evaluation is a dynamic process. The questions which 
can and should be asked vary as a function of where the program is in 
its Life Cycle., Some questions are important only if they can be 
answered before the program becomes well established, that is, during 
implementation. Other questions can be answered only after "the program 
has been used for some period of time. The Life Cycle concept provides 
a framework that guides the evaluator in asking the right questions at 
the right time. 

The use evaluation begins after the program is already established 
in the user's environment. 



Life Cycle Evaluation of Army Training Programs 
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The subissues it addresses are fidelity, sufficiency, and effectiveness. 

\ 

Fidelity evaluation (Fullan & Pomfret, 1977) is a comparison of the 
user's procedures against the developer's ideal. Sufficiency (The area 
of sufficiency evaluation has been referred to by Leinhardt, 1980, as 
Domain-of- Instruction. J focuses on those areas where the user's pro- 
cedures are different from the developer's. The basic concern here is 
whether the new user-generated procedures fulfill the same training 
function as the developer's procedures. Lastly, effectiveness evalua- 
tion assesses the program's actual effectiveness when used routinely in 
the user's environment (as opposed to its effectiveness in the pre- 
fielding evaluation conducted by the developer). 

Implementation monitoring focuses on the process of implementing a 
new training program. The process of implementation begins before the 
program is used at all and its goal is to ensure the routine use of the 
new program. The goal of monitoring this process is to increase its , 
effectiveness. The distinction between the goals of the implementation 
monitor and the goals of the implementation process is important to 
keep in mind. 
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While acting as an implementation monitor, the evaluator is not/ 
interested in whether the program is being used; s/h^ is interested in 
the necessity of planned actions, whether the execution of planned 
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actions accomplishes the goal of the plan, and whether the proximal 
goals of the implementation process are being achieved. The rest of my 
talk will focus on implementation monitoring in general and these three 
subissues in particular. ' * 

Implementation Monitoring 

The next slide depicts the relationship between the three issues 
monitored during implementation. 

1 
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An implementation plan can be considered as a set of planned actions. 
The monitor would like to determine whether the plan contains all 
actions necessary for implementing the program and any that are 
unnecessary. When a planned action is executed then an instantiation 
of that plan exists. The monitor would like to know whether the 
instantiation achieved the goals planned for it or whether something 
was lost during the execution. For example ^ many Army training programs 
require new equipment and require that the trainer be able to perform 
some low level maintenance on the equipment. A "planned action 11 might 
be the production of a pamphlet for the trainer on how to troubleshoot 
the equipment. The particular pamphlet that is produced is an instan- 
tiation of this planned action. We can then ask whether this pamphlet 
provides all the information needed to troubleshoot the equipment and » 
whether the reading level and format is appropriate for its intended 
audience. 
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. The ultimate goal of all the planned actions is to get the new 
program used routinely. However, evaluation of routine use typically 
takes place after most implementation activity has ceased. Therefore, 
if we want to monitor likelihood of routine use we have to assess 
whether the implementation process is achieving certain intermediate or 
proximal goals. For example, one proximal* goal might be that the first 
level trainers, the NCOs, possess the knowledge and skills needed to 
teach the new program. Presumably troubleshooting the equipment is one 
of the skills the trainer must have to use the program. 

To monitor the implementation process the first step is to have a 
model of that process. We like the model that Roberts-Gray Just 
described (C. Roberts-Gray, T. Gray, & W. Gray, 1982). 
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The model provides the basis for analyzing the fit between the innova- 
tion and user. With this information, the model yields an analysis of 
changes in organizational arrangements, individual know-how, organiza- 
tion rules, and individual commitment that are required if the innova- 
tion is to "fit" the user. These changes ^become the. proximal goals of 
the implementation process. ^ Finally, for each change the model yields a 
suggested strategy for accomplishing that change. 

/ 
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In evaluating th£ necessity of the implementation plans, the 
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monitor determines whether the planned actions incorporate the model- 
derived startegies and thus seem likely to accomplish the proximal goals 
of the implementation process. The monitor's role at this point is 
to recommend modifications, deletions, and additions to the planned 
actions. 

The execution evaluation examines whether the instantiation of a 
planned action achieved the planned goals . If the evaluator was not 
involved in implementation planning, then part of %he Job here may 
include a post hoc specification of what immediate goals an action was 
supposed to achieve. For example,, in writing training pamphlets, the 
"nice to know" information is often confused with the "must know" 
information. In this example, the evaluator may first have to determine 

s"; 

what is "must have" and what is "nice to kn<4w" information. Then he may 
have to perform a critical reading of the pamphlet to determine if the 
"must know" information is adequately presented. For other actions the 
evaluator may be required to perform a mini-program evaluation. For 

example, a common vehicle for implementation in the Army is to send a 

I 

mobile training team to each post to train the trainers. In this case 
the evaluator would assess whether course graduates can indeed use the 
program to train others: 

/ 
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The monitor's role in the execution evaluation is to recommend 
modifications to instantiations which are not achieving their planned 
goals. 

The third issue the monitor considers is whether the changes 
required for the innovation to fit the user have taken place, that is, 
whether the implementation process is achieving its proximal goals. The 
monitor's role here is for recommendations for remedial actions . If, 
to go back to the slide, the instantiations of actions Al, A2 and A3 
have achieved their planned goals but proximal goal PI is not achieved, 
then an additional action should be considered. 
Conclusions & Perspectives 

Our experiences with Army training programs has led us to the 
belief that attention to the process of implementation is vital if a 
program is to become a routine part of unit training. ARI at Monterey 
has developed guidance for implementation planners and a framework for 
evaluation. The framework, which I have presented, is based upon 
lessons learned from monitoring implementation efforts. Three sets of 
evaluation issues are considered: the necessity of planned actions, the 
execution of those actions, and their proximal effect. The framework is 
client oriented. It organizes the evaluation issues in terms and 
categories attuned to the political realities and training issues with 
which the client is familiar. Use of the framework requires a model of 
the implementation process; however, it is our intent that while the 
theoretical concepts from the implementation model are used by the 
evaluator, they are functionally "invisible" to Army decision makers. 
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SUBISSUES IN IMPLEMENTATION -MONITORING 
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CHANGES REQUIRED IN: 

- ORGANIZATIONAL ARRANGEMENTS 

- INDIVIDUAL KNOW-HOW 

- ORGANIZATION RULES 

' - INDIVIDUAL COMMITMENT 

FOR THE INNOVATION TO 
"FIT" THE USER 



STRATEGIES FOR ACCOMPLISHING 
PROXIMAL GOALS 

- ASSISTANCE 

- EDUCATION 

- POWER 

- PERSUASION 
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